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content words, associative words associated with the content 
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FIG. 6 
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1 

INFORMATION SEARCH APPARATUS AND 
METHOD, AND COMPUTER READABLE 
MEMORY 

BACKGROUND OF THE INVENTION ^ 

The present invention relates to an information search 
apparatus and method for managing a plurality of kinds of 
multimedia information, and searching the managed multi- 
media information for desired multimedia information, and lO 
a computer readable memory. 

As a conventional information search apparatus which 
searches muliimedia infonnation, e.g., imagps, an informa- 
tion search apparatus that obtains a required image by 
matching an input keyword with that corresponding to an 
image has been realized. Furthermore, an information search 
apparatus, which obtains an image, that cannot be obtained 
by full-word matching with an input keyword, by matdiing 
not only the input keyword but also an associated keyword 
associated with the input keyword with a keyword corrc- ^ 
spending to an image, has also been realized. Moreover, an 
information search apparatus which obtains an image with 
similar color information by detecting a correspondence 
between the input keyword and color information using, 
e.g., color information of images is proposed. 23 

However, with (he conventional information search 
apparatus, the operator cannot designate query conditions 
for obtaining required search results with respect to a search 
request indicated by the input keyword, and cannot obtain 
search results he or she actually intended. For example, 
when a search request "simple" is input, and an "advanced" 
image such as a rocket launching scene that gives an 
impression contrary to "simple" is managed among images 
having quiet color tone and giving "simple" impression", 
such image is presented as an im^e that matches the search 
request "simple". Similarly, when an image having "sbowy" 
color information as color tone contrary to "simple" is 
managed among images having keywords that match an 
associative word "handcraft" of "simple", such image is 
presented as an image that matches the seardi request ^ 
"simple". 

SUMMARY OF THE INVENTION 

The present invention has been made in consideration of 
the above-mentioned problems, and has as its object to 
provide an information search apparatus and method which 
can make an appropriate information search according to the 
designated search request, and a computer readable memory. 

In order to achieve the above object, an information 
search apparatus according to the present invention com- 
prises the following arrangement. 

That is, there is provided an information search apparatus 
for managing a plurality of kinds of multimedia information, 
and searching the managed multimedia information for 
desired multimedia infomiation, comprising: 

an image content word holding unit for storing the mul- 
timedia information, and content words which verbal- 
ize concepLs expressed in the muliimedia information 
in correspondence with each other; 
an associative word dictionary for storing the content 
words and non-associative words which do not evoke 
bilateral association with the content words in corre- 
spondence with eadi other; 
input means for inputting a query word; 65 
first search means for acquiring a non-associative word 
corre^onding to the query word input by the input 
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means from the associative word dictionary, and 
searching multimedia information on the basis of the 
acquired non-associative word; 
second search means for extraaing a feature amount 
corresponding to the query word input by the input 
means, and searching multimedia information on the 
basis of the extracted feature amount; and 
integration means for integrating first and second search 
results obtained by the first and second search means on 
the basis of the first search results. 
In order to achieve the above object, an information 
search method according to the present invention comprises 
the following arrangement. 

That is, there is provided an information search method 
for managing a plurality of kinds of multimedia information, 
and searching the managed multimedia information for 
desired multimedia information, comprising: 

the storage step of storing on a storage medium an image 
content word holding unit for storing the multimedia 
information, and content words which verbalize con- 
cepts expressed in the multimedia information in cor- 
respondence with each other, and an associative word 
dictionary for storing the content words and non- 
associative words which do not evoke bilateral asso- 
ciation with the content words in correspondence with 
each other; 
the input step of inputting a query word; 
the first search step of acquiring a non-associative word 
corresponding to the query word input in the input step 
from the associative word dictionary, and searching 
multimedia information on the basis of the acquired 
non -associative word; 
the second search step of extracting a feature amount 
corresponding to the query word input in the input step, 
and searching multimedia information on the basis of 
the extracted feature amount; and 
the integration step of integrating first and second search 
results obtained in the first and second seardi steps on 
the basis of the first search results. 
In order to achieve the above object, a computer readable 
memory according to the present invention comprises the 
following arrangement. 

That is, there is provided a computer readable memory 
which stores a program code of an information search 
process for managing a plurality of kinds of multimedia 
information, and searching the managed multimedia infor- 
mation for desired multimedia information, comprising: 
a program code of the storage step of storing on a storage 
medium an image content word holding unit for storing 
the multimedia information, and content words which 
verbalize concepts expressed in the multimedia infor- 
mation in correspondence with each other, and an 
associative word dictionary for storing the content 
words and non-associative words which do not evoke 
bilateral association with the content words in corre- 
spondence with each other; 
a program code of the input step of inputting a query 
word; 

a program code of the first seard] step of acquiring a 
non-assodative word corresponding to the query word 
input in the input step from the associative word 
dictionary, and searching multimedia information on 
the basis of the acquired non-associative word; 

a program code of the second search step of exuacting a 
feature amount corresponding to the query word input 
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in the input step, and searching multimedia infonnalioo 
on the basis of the extracted feature amount; and 
a program code of the integration step of integrating first 
and second search results obtained in the first and 
second search steps on the basis of the first search 
results. 

In order to achieve the above object, an infonnation 
search apparatus according to the present invention com- 
prises the following arrangement. 

That is, there is provided an information search apparatus 
for managing a phirality of kinds of multimedia information, 
and searching the managed multimedia information for 
desired multimedia infonnation, comprising: 

an image oootent word holding unit for storing the mul- 
timedia information, and content words which verbal- 
ize concepts expressed in the multimedia information 
in correspondence with each other; 
an associative word dictionary for storing the content 
words, associative words associated with the content ^ 
words, and non-associative words which do not evoke 
bilateral association with the content words in corre- 
spondence with each other; 
input means for inputting a query word; 
first scardi means for acquiring an associative word 25 
corresponding to the query word input by the input 
means from the associative word dictionary, and. 
searching multimedia information on the basis of the 
acquired associative word; 
second search means for acquiring a non-assodative word 
corresponding to the query word input by the input 
means, and searching multimedia information on the 
basis of the acquired non-associative word; and 
integration means for integrating first and second search 
results obtained by the first and second search means on 
the basis of the first search results. 
In order to achieve the above object, an information 
search method according to the present invention comprises 
the following arrangement. 

That is, there Is provided an information search method *° 
for managing a plurality of kinds of multimedia information, 
and searching the managed multimedia information for 
desired multimedia information, comprising: 

the storage step of storing on a storage medium an image 
content word holding unit for storing the multimedia 
information, and content words which verbalize con- 
cepts expressed in the multimedia information in cor- 
respondence with each other, and an associative word 
dictionary for storing the content words, associative 
words associated with the content words, and non- 
associative words which do not evoke bilateral asso- 
ciation with the content words in correspondence with 
each other; 
the input step of inputting a query word; 
the first search step of acquiring an associative word 
corresponding to the query word inpui in the input step 
from the associative word dictionary, and searching 
multimedia information on the basis of the acquired 
associative word; 
the second search step of acquiring a non-assodative 
word corresponding to the query word input in the 
input step, and searching multimedia information on 
the basis of the acquired non-associative word; and 
the integration step of integrating first and second search 
results obtained in the first and second search steps on 
the basis of the first search results. 
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In order to achieve the above object, a computer readable 
memory according to the present invention comprises the 

following arrangement. 

That is, there is provided a computer readable memory 
which stores a program code of an information search 
process for managing a plurality of kinds of multimedia 
information, and searching the managed multimedia infor- 
mation for desired multimedia information, comprising: 
a program code of the storage step of storing on a storage 
medium an image content word holding unit for storing 
the multimedia information, atxl content words which 
verbahze concepts expressed in the multimedia infor- 
mation to correspondence with each other, and an 
associative word dictionary for storing the content 
words, associative words associated with the content 
words, and non-associative words u^ch do not evoke 
bilateral association with the content words in corre- 
spondence with each other; 
a program code of the input step of inputting a query 
word; 

a program code of the first search step of acquiring an 
associative word corresponding to the query word input 
in the input step from the associative word dictionary, 
and searching multimedia information on the basis of 
the acquired associative word; 

a program code of the second search step of acquiring a 
non-associative word corresponding to the query word 
input in the input step, and searching multimedia infor- 
mation on the basis of the acquired no n- associative 
word; and 

a program code of the integration step of integrating first 
and second search results obtained in the first and 
second search steps on the basis of the first search 
results. 

Other feahires and advantages of the present invention 
will be apparent from the following descrq>tion taken in 
conjunction with the accompanying drawings, m which like 
reference characters designate the same or similar parts 

throughout the figures thereof. 

BRIEF DESCRIPTION OF THE DRAWINGS 

FIG. 1 is a block diagram lowing the arrangement of an 
information search apparatus according to an embodiment of 
the present invention; 

FIG. 2 is a block diagram showing the functional anange- 
ment of the information search apparatus according to the 
embodiment of the present invention; 

FIG. 3 is a table showing the structure of a sensory 
pattern/associative word correspondence holding unit in the 
embodiment of the present invention; 

FIG. 4 is a table showing the sUucturc of an unfoWed 
sensory pattern holding unit in the embodiment of the 
present invention; 

FIG. 5 is a table showing the stmcture of a sensory pallem 
holding unit in the embodiment of the present invention; 

FIG. 6 is a table showing an example of image feature 
amounts in the embodiment of the present invention; 

FIG. 7 is a table showing the structure of an image feature 
amount & sensory pattern holding unit in the embodiment of 
the present invention; 

FIG. 8 shows a display example of a search perspective 
input by a search request input processing unit in the 
embodiment of the present invention; 

FIG. 9 shows a display example on a control panel upon 
instnicling search weights in the embodiment of the present 
invention; 
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FIG. 10 is a t^le showing the structure of an image result holding unit 216, «4udi will be described later with 

holding unit in the embodiment of the present invention; reference to FIG. 2. 

FIG. 11 is a table showing the structure of an image Reference numeral 14 denotes an external memory 

content word holding unit in the embodiment of the present (DISK), which stores a concept discrimination dictionary 

invention; 5 205, associative word dictionary 211, image word/sensory 

FIG. 12 is a table showing another example of the paiiera correspondence holding unit 215, image content 

structure of an image content word holding unit in the word holding unit 219. image holding unit 218. sensory 

embodiment of the present invention; pattern holding unit 220, image feanire amount holding unit 

BG. 13 is a table showing the stiuctuic of a concept 222. and unagc feamie amoimt/scnsory pattern correspon- 

discriminalion dicUonary in the embodiment of the present " »^°^*^°y?^223 which will be described later with 

invention- reference to FIG. 2. Also, the external memory 14 stores 

„^ , ' ..... , . . programs for respectively implementing processing units, 

FIG. 14 .5 a table showing the structure of an assoaauve .^^ ^ ^^^^ ^ processing unit 201, weight 

word dicUonary in Ihe emboduncnt of the present mvention; determination processing unit 206, associative word unfold- 

FIG. 15 is a table showing the structure of a search result IS ing proce^g unit 208, image content word search unh 210 

holding unit in the embodiment of the present invention; using associative words, sensory pattern unfolding proccss- 

FIG. 16 is a table showing another example of feature ing unit 212, sensory pattern search processing unit 214, 

amounts in the embodiment of the present invention; search result integration processing unit 217, image feature 

FIG. 17 is a table showing the structure of an image amo^m extraction processing unit 221, and sensory pattern 

feature amount holding unit in the embodiment of the ^ determination processing unit 224, which will be described 

present invention- reference to FIG. 2. As a storage medium for 

FIG. 18 is a flow chart showing processes executed in the ^^^^ P^°S^^^^' ^. ^^^^l ^ ^^I'^^^J* 

embodiment of the present invention; '"'^"^^^ magnetoopUcal disk, or the like can be used. 

FIG. 19 is a flow chart showing details of a search request 25 , R«f«^P<» ^^^^^^^ 15 denotes a keyboard (KB) which has 

input process in the embodiment of the present invention; t ^^'^'^ ^'^y^' ^'^^^^^ ^^y^' " ^y™" 

' ' . „ „ . bol mput keys for mputlmg a penod, comma, and the like, 

HG. 20 IS a flow chartjhowngdetaiU of a search process ^ ^^^^^ ^ insirucUng a search, and various function 

using an associative word in the embodunent of the present ^^^^ ^^^^ ^ ^^^^ ^^^g ^^^^ instructing cursor 

mvention; movemcni, and the like. Reference numeral 16 denotes a 

FIG. 21 is a flow chart showing details of a search process 30 display video memory (VRAM) for storing a pattern of data 

using a sensory pattern in step S3006 and a search result to be displayed. Reference numeral 17 denotes a CRT 

combining process in step S3007 in tiie embodiment of the controUcr (CRTC) for displaying the contents stored in the 

present invention; and VRAM 16 on a CRT 18. Reference numeral 18 denotes a 

FIG. 22 is a flow chart showing an image registration display device (CRT) using, e.g., a cathode ray tube, or the 

process in the embodiment of the present invention. like. The dot display pattern and cursor display on the CRT 

18 arc controlled by the CRTC 17. Note that various other 

DESCRIPTION OF THE PREFERRED displays such as a hquid crysUl display, and the like may be 

EMBODIMENTS as tiie display device. Reference numeral 19 denotes a 

The preferred embodiments of the present invention wiU network controller (NIQ, which connects the apparatus to a 

be described in detail hereinafter with reference to the network such as Ethernet or the like, 

accompanying drawings. The information search apparatus constructed by the 

FIG. 1 shows the arrangement of an information search aforementioned building components operates in accor- 

apparatus according to an embodiment of the present inven- dance with various inputs from the KB 15 and various inputs 

tion. 45 supplied from the NIC 19 via the network. Upon receiving 

Referring (o FIG. 1, reference numeral 11 denotes a *e input from die KB 15 or NIC 19, an inlcrmpt signal is 

microprocessor (CPU), which makes computations, logical to the CPU 11. the CPU 11 reads out various control 

decisions, and the like for image information search, and signals stored in the DISK 14, and executes various kinds of 

controls individual building components connected to an control in accordance with these conUr)! signals. Also, the 

address bus AB, control bus CB, and data bus DB via these 50 P''^^^"' invention is achieved by supplying a storage 

buses. The address bus AB transfers an address signal medium that stores a program according to die present 

indicating the building component to be controlled by the invention to a system or apparatus, and by reading out and 

CPU 11 . The control bus CB transfers and appUes a control executing program codes stored in die storage medium by a 

signal for each buUding component to be controlled by the computer of the system or apparatus. 

CPU 11. The data bus DB transfers data among the respec- ss functional arrangement of the information search 

tive building oompooents. apparatus of Ibis embodiment will be explained below with 

Reference numeral 12 denotes a read-only memory reference to FIG. 2. 

(ROM), which stores control programs such as a processing FIG. 2 is a block diagram showing the functional arrange- 

program executed in this embodiment, and the like. Refer- meni of the information search apparatus according to Ihe 

ence numeral 13 denotes a rewritable random access 60 embodiment of the present invention. Referring to FIG. 2, 

memory (RAM) which is configured by 16 bits per word, reference numeral 201 denotes a search request input pro- 

and is used as a temporary storage of various data from the ccssing unit for inputting query items (query word, search 

respective building components. Also, the RAM 13 stores a perspective or category, search weight, and the like) that 

query word holding unit 202, search perspective holding pertain to the information wanted. Reference numeral 202 

unit 203, search weight holding imit 204, determined weight 65 denotes a query word holding unit for storing a query word 

holding unit 207, unfolded associative word holding unit input by the search request input processing unit 201. 

209, unfolded sensory pattern holding unit 213, and search Reference numeral 203 denotes a seardi perspective holding 
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unit for storing a search perfective input by the search gniic^ the search results of image content words using the 

request input processing unit 201. Reference numeral 204 associative words, and the search results of sensory patterns 

denotes a search weight holding unit for storing a search stored in a search result holding unit 216, on the basis of the 

weight input by the search request input processing unit 201. stuch weights obtained by the weight detennination pro- 

Rcfcrcncc numeral 205 denotes a concept discrimination S ceding unit 206. 

dictionary having a search perspective that pertains to a ^ Reference mimeral 219 denotes an image content word 

concept as the informaUon wanted, an antithetic concept l^^^.^-^g verbalmng and stonng concepts express^ 

having a contrary or antonymous meaning, and two kinds of ^ ^*S« information ^ored in an image-holdmg unit 218 

_° . r ■ . . J- • ■ .• r Reference numeral 218 denotes an image holding unit for 

coeffiacn^ for weight discnmmauon upon searching for a ^^^^ . informaUon serving as test images. Reference 
concept. Reference mxmeral 206 denotes a weight determi- lo ^^^^^ ^20 denotes a sensory pattern holding unit for 

nauon processing unit for giving weights mdicaUng the folding sensory patterns obtained from Ihc image informa- 
weight balance on associative words or sensory patterns jjoj^j (he image holding unit 218, and storing 

upon searching using a query word stored in the query word matching levels with respective sensory patterns in units of 

holding unit 202. Reference numeral 207 denotes a deter- image IDs each indicating image information. FIG. 5 shows 

mined weight boldii:^ unit for holding the search weight adatastorageexampleof the sensory holding unit 220. Note 

determined by the weight determination processing unit 206. that the structure of the sensory pattern holding unit 220 wfll 

Reference numeral 208 denotes an associative word be described in detail later, 
unfolding processing unit for unfolding the query word Reference numeral 221 denotes an image feature extrac- 
storcd in the query word holding unit 202 into associative tion processing unit for extracting physical image feature 
words with reference to an associative word dictionary 211, amounts from image information stored in the image hold- 
obtaining an antithetic concept antonymous to the input ing unit 218. Physical image feature amounts arc visual 
query word from the concept discrimination dictionary 205, features or signatures extracted from regions segmented on 
and unfolding the antithetic concept into associative words. the basis of color information, blocks each segmented to 
Reference numeral 209 denotes an unfolded associative have a given area, or the entire image. The image feature 
word holding unit for holding the associative words ^ amount is, e.g., numerical information such as the color 
unfolded by the associative word unfolding processing unit distribution or histogram, density distribution, texture, edge, 
208. Reference numeral 210 denotes an image content word frequency distribution, and the like of an image, and is 
search processing unit using associative words, which finds expressed, as shown in, e.g., FIG. 6. Note that the image 
image content words, which are stored in an image content feature amounts will be described in detail later, 
word holding unit 219 and match the unfolded associative Reference numeral 222 denotes an image feature amount 
words, by search with reference to the unfolded associative holding unit for storing the image feature-amounts obtained 
word holding unit 209. by the image fcamrc amount extraction processing unit 221. 

Reference numeral 211 denotes an associative word die- Reference numeral 223 denotes an image feature amount/ 

tionary for storing associative words to be unfolded in units sensory pattern correspondence holding unit for storing 

of concepts serving as index words in correspondence with image feature amounts and sensory patterns in correspon* 

associative perspectives. Reference numeral 212 denotes a dence with each other, i.e., storing sensory pattern IDs and 

sensory pattern unfolding processing unit for unfolding the image feature amounts corresponding to those IDs. FIG. 7 

query word stored in the query word holding unit 202 into shows a data storage example of the image feature amount/ 
sensory patterns with reference to an image word/sensory ^ sensory pattern correspondence holding unit 223. Note that 

pattern correspondence holding unit 215, obtaining an anti- the image feature amounl/sensory pattern correspondence 

ihetic concept antonymous to the stored query word from the holding unit 223 will be described in detail later, 
concept discrimination dictionary 205, and unfolding the Reference numeral 224 denotes a sensory pattern deter- 

obtained antithetic concept into sensory patterns. mination processing unit for comparing a sensory pattern 

Reference numeral 215 denotes an image word/sensory 45 and image feature amount extracted from image information 

pattern correspondence holding unit for storing image woitls to obtain their matching level with reference to the image 

and sensory patterns in correspondence with each other, i.e., feature amount/sensory pattern correspondence holding unit 

storing image words and sensory pattern IDs corresponding 223. 

to associative words, which are associated with the image A display example of a search perspective that pertains to 
words. FIG. 3 shows a data storage example of the image 50 search request items input at the search request input pro- 
word/sensory pattern correspondence holding unit 215. Note cessing unit 201 will be explained below with reference to 
that the structure of the image word/sensory pattern corre- FIG. 8. 

spondence holding unit 215 will be described in detail later. FIG. 8 shows a display example of a search perspective 

Reference numeral 213 denotes an unfolded sensory pattern input at the search request input processing unit in the 
holding unit for temporarily storing the sensory patterns 55 embodiment of the present invention, 
unfolded by the sensory pattern unfolding processing unit FIG. 8 illustrates that three search perspectives "color 

212. The unit 213 is stored in the RAM 13. FIG. 4 shows a tone", "taste", and "general atmosphere" are available in 

data storage example of the sensory pattern unfolding pro- relation to a query word "mild", and hatched "color tone" is 

cessing unit 213. Note that the structure of the sensory selected as the search perspective. By pressing one of the 

pattern unfolding processing unit 213 will be described in cursor moving keys on the keyboard 15, the hatehing moves 

detail later. from "color tone" to "taste" or "general atmosphere", and 

Reference numeral 214 denotes a sensory pattern search Ihe user can designate a desired search perspective or 

processing unit for finding sensory patterns, which are category. 

stored in the sensory pattern holding unit 220 and are similar A display example on the control panel when the operator 
to the unfolded sensory patterns, by search with reference to 65 instructs the search weight balance on a search using asso- 

the sensory pattern holding unit 220. Reference numeral 217 ciative words and a search using sensory patterns in actual 

denotes a search result integration processing unit for inte- search will be explained bebw with reference to RG. 9. 
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FIG. 9 shows a display example of the control panel upon associated with index words 2050, antithetic concepts 2052 

instructing search weights in the embodiment of the present having meanings contrary to the index words 2050, assocd- 

invention. ated weights 2053 used upon searching the index words 

Referring lo FIG. 9, when the user slides a slide button 41 2050, and sensory pattern weights 2054 used upon searching 
to the left, an instruction that sets a heavier weight on a 5 the index words 2050 in correspondence with each other, 

search using associative words is issued; when he or she The structure of the associative word dictionary 211 will 

slides the slide button 41 to the right, an instruction (hat sets be explained below with reference lo FIG. 14. 

a heavier weight on a search using sensory patterns is issued. FIG. 14 shows the stnicture of the associative word 

When the user designates search weights using the slide dictionary in the embodiment of the present invention, 

button 41 and then presses a button 43, a search weight « associative word dictionary 2U is composed of 

instruction is issued. associative IDs 2110 each of which assigns a imique munber 

The structure of the image holding unit 218 wiU be to a set of associative words for each index word 2111, index 

described below using FIG. 10. words 2111 each serv^ing as a start point of association, 

FIG. 10 shows the strucuire of the image holding unit in . associative words 2112 evoked by the index words 2111, 

the embodiment of the present invention. associative perspectives 2113 which arc relevant to associa- 

-nie image holding unit 218 manages image informaUon ^^^^^ °^ ^^e associative words 2112, and association 

by storing image IDs each indicating image informaUon strengths 2U4 each mdicatmg the strength of associaUon 

(image files) and image file storage paths each indicating the ^^^^^^ ^^^^ ^"""^ ^^^^^^^^^ 

storage location of image information. Referring to FIG. 10. TTw^assoaation strength 2114 assumes an absolute 

reference numeral 2180 denotes an image ID which is ^^^"^ ^S^S ° ^'^n indicates direction 

uniquely assigned to one image file. Reference numeral of associaUon of the associaUve word. More specificaUy, 

2181 denotes a file path which indicates the storage location associaUon strength is a posiUve value, it indicates 

of an image file corresponding to the image ID in the DISK * ^t^°°g", associative relationship higher bilateral 

14. and corresponds lo the directory and file of MS-DOS. An 25 associaUon strength value is larger; when 

image file is divided into header and image data fields (not ihe associaUon strength is a negauve value, il indicates a 

shown in no. 10). The header field stores information harder associaUve relatioiiship as the associaUon strength 

required for reading data from that image file, and additional ^^."f ;f ^^S^^- .^^^^P^^' an assocauve word '^folkcraf l 

information that explains the image contents. As such corresponding to an index word smiplc in asso- 
information, an image format identifier indicating the image 30 cuf « djila with the assoaative ID- 126533 can be assoa- 

format name of the image, file size, image width, height, and "^'^ ^ , but an associaUve word chandeher 

depth, the presence/absence of compression, color pallet associative data with the associaUve lD-126536 is hardly 

information, resolution, ofisct lo the storage location of associated with strength "9" smcc its association strength is 

image data, and the Uke are stored. The image data field * negaUve value. 

stores image data in turn. This embodiment uses the BMP 35 The structure of the search result holding unit 216 will be 

format of Microsoft Corp. as such image format, but other described below with reference to FIG. 15. 

compression formats such as GIF, JPEG, FlashPix, and the FIG. 15 shows the structure of the search result holding 

like may be used. unit in the embodiment of the present invention. 

The structure of the image content word holding unit 219 The search result holding unit 216 stores image IDs which 
will be described below wilh the aid of FIG. 11. 40 arc found by searches of the image content word search 

FIG. U shows the stnicture of the image contcm word processing unit 210 using associaUve words and the sensory 

holding unit in the embodiment of the present invention. P^^^^"^ search processing unit 214. Rcfcmng to FIG. 15, 

_ . . , jv,.. • reference numeral 2 1 60 denotes a field for Storing image IDs 

The image content word holding unit 219 manages miage ^^^^^ ^ ^^^^^ 2I6I, a field for storing the number of 

informaUon by storing lh^ «nage IDs and miage content ^^^^^^J associative words with posilive association 

words m corresponderice with each other. Referring to HG. ^^^^^^ ^^^^ ^^^^^^ processing unit 

11, reference numeral 21900 denotes a field for slonng -,1-. • • j j n^-^ a ^A e . • 

. ' „ J- . .1. • DA I. • 217 using associative words; and 2162, a field for storing a 

image IDs corresponding to the image IDs 2180 shown in . . ... . . iT j 

cir- 1ft j"»inm fi i^ f . • ^ hst a maximum of 20 associative word IDs 2110 of matched 

FIG. 10; and 21901, a field for stonng image content words ... j • .u • j j- in 

. ' ' . J- . .L • im associative words in the associative word dictionary 211. 

^A^'.^""". ^"^'Sc nies corresponding to tne miage lus ^j^^ ^^^^^ ^161 of matched associative words is 

21900. The miage content word verbalizes an image feature associative ID 2162 is filled with IWLL code, 

expressed m ari image file and stores a keyword as a ^^^^^^^ 2163 denotes a field for storing the 

character code (e g-.;ji"«>de)^ A plurahty of keywords may ^ associative words with respect to 

be siored perimage file, and the image con en word hol^^^ ,^^^2160. When the number 2161 of mVlched 

unit 219 is expressed as a iLSt of image content words 21901 . .? . . ... * l- i i 
^intxn 1 u ■ it .u ss associalivc words IS zero, the assoaative matoiing level 

usingimageIDs2l900askeys. Or.asshowninFlG. 12, the ^^ci » 

. . J L »j' u 1 21fi3 Stores zero, 

image content word boldmg unit 219 may be expressed as a _ , c ■ ^ 

listof imageIDs21911 using image conlenl words 21910 as Reference numeral 2164 denotes a field for stonng the 

c -o number of sensory patterns with highest similarity, which 

........ are found by search by ihe sensory pattern search processing 

The stractiue of the concept d^rinjmaUo^ 60 unit 223; and 2165. a field for storing a list of a maximum 

wiU be descnbed below usmg FIG. 13. 20 sensory paUern IDs of matched sensory patterns. When 

HG. 13 shows the structure of the concept discrimination the number 2164 of matched sensory paUems is zero, the 

dictionary in the embodiment of the present invention. sensory pattern ID 2165 is filled with NUlXcode. Reference 

As shown in FIG. 13, the concept discrimination dictio- numeral 2166 denotes a field for storing the search matching 
nary 205 provides information that pertains to a query word 65 level of a sensory pattern search with respect to the image ID 

serving as a search request, and stores index words 2050 2160. When the number 2164 of matched sensory patterns is 

corresponding to query words, search perspectives 2051 zero, the sensory pattern matching level 2166 stores zero. 
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Reference numeral 2167 denotes a field for sioring ihe 
matching level of th e image ID 2160 with respect to the 
search request, which is calculated using the associative 
matching level 2163 and sensory pattern matching level 
2166 as parameters. 

The structure of the above-mentioned unfolded sensory 
pattern holding unit 213 will be described below with 
reference to FIG. 4. 

Referring to FIG. 4, reference numeral 2130-1 denotes aa 
image word as an unfolding source from which this sensory 
pattern has evolved upon unfolding, and the same image 
word as that in the query word holding unit 202 is stored In 
this embodiment, a character string "refreshing^ is stored, 
and ends with NULL code. Reference numeral 2130-2 
denotes the number of sensory patterns unfolded from the 
image word "refreshing'*, and the number of sensory pat- 
terns is 7 in this embodiment. Reference numeral 2130-3 
denotes an address indicating the storage location area of 
data obtained by actually unfolding the image word 
"refreshing". The address 2130-3 is linked with unfolded 
data 2130-4. 

Reference numeral 2130-4 denotes unfolded data actually 
unfolded from "refreshing", and sets of associative words 
and sensory patterns corresponding to the number 2130-2 of 
sensory patterns are stored here. In this embodiment, seven 
sets of associative words and sensory patterns are stored. 
Reference numeral 2130-5 denotes a sensory pauern ID 
corresponding to the image word "refreshing" and an asso- 
ciative word "forest" of "refreshing". In this embodiment, 
"5" is stored. Reference numeral 2130-6 denotes an asso- 
ciative word of the image word "refreshing". In this 
embodiment, a character string "forest" is stored, and ends 
with NULL code. 

The structure of the aforementioned image word/sensory 
pattern correspondence holding unit 215 will be described in 
detail below using FIG. 3. 

Referring to FIG. 3, reference numeral 2150-1 denotes an 
image word serving as an unfolding source of this sensory 
pattern. In this embodiment, character strings "refreshing", 
"tropical", and the like are stored, and end with NULL code. 
Reference numeral 2150-2 denotes an associative word 
unfolded from the image word 2150-1. In this embodiment, 
associative words "forest", "tableland", "blue sky", and the 
like are stored in correspondence with "refreshing^', and 
these character strings end with NULL code. When no 
character siring is stored in this field, i.e., NULL code alone 
is stored, this sensory pattern applies to all image words 
"refreshing"; no specific associative word has been desig- 
nated. 

Reference numeral 2150-3 denotes a sensory pattern ID 
corresponding to the image word 2150-1 and associative 
word 2150-2. In this embodiment, "005" and "006" are 
stored as sensory pattern IDs corresponding to the image 
word "refreshing" and its associative word "forest". Also, 
sensory pattenns for "not refreshing" as an antilbetic concept 
of "refreshing" are stored. In this embodiment, for "not 
refreshing", no associative words are registered and "001" 
and "010" arc registered as sensory pattern IDs. 

The structure of the above-mentioned sensory pattern 
holding unit 220 will be described in detaQ below using HO. 
5. 

Referring to FIG. 5, reference numeral 2200-1 denotes an 
image ID for identifying an image to be registered. The 
image IDs use the same ones as those stored in the image 
holding unit 218, and uniquely define images in this system. 
A field 2200-2 stores sensory pattern IDs. In this 
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embodiment, since the matchmg levels between each image 
and all sensory patterns stored in the image feature amount/ 
sensory pattern correspondence holding unit 223 are 
calculated, all the sensory pattern IDs arc stored. Reference 

s numeral 2200-3 denotes a numerical value indicating the 
matching level between each image and sensory pattern. The 
matching level assumes a value ranging from 0 to 1; 0 
indicates the image docs not match the sensory pattern at all, 
and the matching level becomes higher as it is closer to 1, 

10 For example, the matching level between image with the 
image ID-001 aixl sensory pattern 1 is 0.10, aixl the 
matching level between that image and sensory pattern 2 is 
0. 

The aforementioned image feature amounts will be 
15 explained in detail below with reference to FIG. 6. 

In FIG. 6, XI, X2, X3, . . . , Xn represent image features 
extracted from one image, Bl, B2, . . . , Bm represent 
regions/blocks from which image feature amounts are 
extracted, and xll to xmn represent image feature amounts 
20 extracted from the individual regions/blocks. FIG. 16 exem- 
plifies a case wherein chromatic image feature amounts are 
exiracied, i.e., representative colors are extracted in units of 
image regions or blocks. Representative colors extracted 
from regions/blocks Bl, B2, . . . , Bn are C1(R1, Gl, Bl), 
^ C2(R2, G2, B2), . . . , Cn(Rn, Gn, Bn), and their image 
feature amounts are cl to cn. 

The structure of the image feature amount holding imit 
222 will be described below using FIG. 17. 

FIG. 17 shows the structure of the image feature amount 
holding unit in the embodiment of the present invention. 

Referring to FIG. 17, reference numeral 2220-1 denotes 
an image ID for identifying an image to be registered. The 
image IDs use the same ones as those stored in the image 
holding unit 218. Reference numeral 2220-2 denotes a block 
or region number from which an image feature amount is 
extracted. In this embodiment, Bl, B2, . . . , 6m represent 
the region/block numbers. Reference numeral 2220-3 
denotes information indicating an image feature extracted 
^ from each of the regions/blocks Bl, B2, . . . , Bm (2220-2). 
This embodiment exemplifies a case wherein chromatic 
image features are extracted, and a plurality of pieces of 
information CU(R11, GU, BU), . . . , Cnl(Rnl, Gnl, Bnl) 
indicating colors arc stored. Reference numeral 2220-4 
denotes image feature amounts of image features extracted 
from the individual regions/blocks. In this embodiment, cll, 
. . . , cln arc stored as the image feature amounts of image 
features CU(RU, GU, Bll), .... Cnl(Rnl, Gnl, Bnl). 

The structure of the image feature amount/sensory pattern 
5Q correspondence holding unit 223 will be described in detail 
below using FIG. 7. 

Referring to FIG. 7, reference numeral 2230-1 denotes a 
sensory pattern ID, which uniquely identifies a .sensory 
pattern. Reference numeral 2230-2 denotes an image feature 
55 amount corresponding to each sensory pattern ID. In this 
embodiment, a sensory pattern is expressed by a chromatic 
image feature amount, and a combination of color compo- 
nents (values in a color qiace such as RGB, HVC, or the 
like) corresponding to each sensory pattern ID is stored. In 
this embodiment, values in the RGB color ^aoe are regis- 
tered as color components. The RGB values assume integers 
ranging from 0 to 255, and a maximum of m colors 
correspond to each sensory pattern ID. 

The processes executed in this embodiment will be 
65 described below using FIG. 18. 

FIG. 18 is a flow chart showing the processes executed in 
the embodiment of the present invention. 
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In step S3001, a processing module that implcmeats the 
operatioD of the search request input processing unit 201 in 
FIG. 2 executes a search request input process. Note that the 
search request input process will be explained in detail later. 
If it is determined with reference to the contents of the 
search weight holding unit 204 in step S3002 that search 
weights are designated, the designated values are stored in 
the determined weight holding unit 207. On the other hand, 
if no search weights are designated, index words 2050 are 
searched for a query word stored in the query word holding 
unit 202 with reference to the concept discrimination dic- 
tionary 205 so as to read out a corresponding associated 
weight 2053 and seasory pattern weight 2054, and the 
readout weights are stared in the determined weight holding 
unit 207. If there is no index word 2050 thai is relevant to 
the contents of the query word holding unit 202, a default 
value "5" is stored as both the associated and sensory pattern 
weights in the determined weight holding unit 207. 

It is checked with reference to the determined weight 
holding unit 207 in step S3003 if the associated weight is 
zero. If the associated weight is zero (YES in step S3003), 
the flow advances to step S3005. On the other hand, if the 
assodated weight is not zero (NO in step S3003), the flow 
advances to step S3004. 

In step S3004, a processing module that implements the 
operations of the associative word unfolding processing unit 
208 and image content word search processing unit 210 
using associative words in FIG. 2 executes a search process 
using associative words. Note that the search process using 
associative words will be described in deuil later. 

It is checked with reference to the determined weight 
holding unit 207 in step S3005 if the sensory pattern weight 
is zero. If the sensory pattern weight is zero (YES in step 
S3005). the flow advances to step S3007. Go the other hand, 
if the sensory pattern weight is not zero (NO in step S3005), 
the flow advances to step S3006. 

In step S3006, a processing module that implements the 
operations of the sensory pattern unfolding processing unit 
212 and sensory pattern search processing unit 214 in FIG. 
2 executes a search process using sensory patterns. Note that 
the search process using sensory patterns will be described 
in detail later. In step S3007, a processing module that 
implements the operation of the search result integration 
processing unit 217 executes a search result integration 
process. Note that the search result integration process will 
be described in detail later. 

In step S3008, image files corresponding to image IDs 
stored as search results obtained in step S3007 are read out 
from the image holding unit 218, and are displayed. Note 
that this process is a known one which is prevalent in image 
search apparatuses of the same type. 

The search request input process in step S3001 will be 
described in detail below with reference to FIG. 19. 

FIG. 19 is a flow chart showing the details of the search 
request input process in the embodiment of the present 
invention. 

In step S2011, a query word serving as a search request is 
input. The query word input is attained by storing a character 
code input at the KB 15 in the query word holding unit 202 
on the RAM 13. In step S2012, search perspectives that are 
relevant to the query word stored in the query word holding 
unit 202 are extracted from the concept discrimination 
dictionary 205. That is, ail search perspectives 2051 corre- 
sponding to index words 2050, which match the query word 
in the query word holding unit 202, are extracted. For 
example, when the query word is "mild", three search 
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perspectives "color tone", "taste", and "general atmosphere" 

can be obtained. 

It is checked in step S2013 if a search perspective or 

perspectives is or are found. If a search perspective or 
5 perspectives is or are found (YES in step S2013), the flow 

advances to step S2014. On the other hand, if no search 

perspective is found (NO in step S2013), the flow advances 

to step S2016. 
In step S2014, the window for designating the search 
^° perspective described above with reference to FIG. 8 is 

displayed. In step S2015, the user selects a desired one of the 

search perspectives displayed on the window. The selected 

search perspective is stored in the search perspective holding 

unit 203. 

In step S2016, the user inputs search weights which 
determine the weight balance on a search using associative 
words and a seardi using sensory pattern in actual search in 
relation to the search process in response to the search 
request. That is, the user operates the slide button 41 on the 

^ control panel shown in FIG. 9 to designate the wei^t ratios 
on associative words and sensory patterns. When the user 
does not designate any search weights, he or she presses the 
button 42 in the display area on the control panel shown in 
FIG. 9 to designate default values of the search weights. 

^ It is checked in step S2017 if search weights are desig- 
nated. If search weights arc not designated (NO in step 
S2017), i.e., if the default values of the search weights are 
designated, the processing ends. On the other hand, if search 

30 weights are designated (YES in step S2017), the designated 
associative word and sensory pattern weights arc stored in 
the search weight holding unit 204 in step S2018, thus 
ending the processing. 

The search process using associative words in step S3004 
will be described in detail bebw with reference to die flow 
chart in FIG. 20. 

FIG. 20 is a flow diart showing the details of the search 
process using associative words in the embodiment of the 
present invention. 

40 In step S2101, associative word data corresponding to 
index words 2111 in the associative word dictionary 211, 
that match the query word stored in the query word holding 
unit 202, are found by search. If index words that match the 
query word are found, all their associative IDs are stored in 

45 the unfolded associative word holding unit 209. In step 
S2102, the concept discrimination dictionary 205 is 
searched, and if an index word that matches the query word 
in the query word holding unit 202 is found, a search 
perspective 2051 corresponding to that index word is 

50 extracted. The extracted search perspective 2051 is com- 
pared with that stored in the search perspective holding unit 
203, and if they tnatch, an antithetic concept 2052 corre- 
sponding to this index word is extracted. On the other hand, 
if the two search per^ectives do not match, dau in which 

55 the query word matdies an index word continues to be 
searched for, and if no antithetic concept whose search 
perspective matches the index word is found finally, the flow 
advances to step S2103. 
In step S2103, the associative word dictionary 211 is 

60 searched for associative words having an index word, which 
matches the antitheu'c concept found in step S2102. If an 
index word that matches the antithetic concept is found, their 
associative IDs are stored in the unfolded associative word 
holding unit 209 by appending a status code indicating an 

65 antithetic concept thereto. 

In step S2104, associative words are extracted based on 
the associative IDs stored in the unfolded associative word 
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holding unit 209, and the image contenl ward holding unit holds associative words "forest", 'Habieland", "bhie sky", 

219 is searched for image content words that match the and the like unfolded from "refreshing", and the image word 

associative words. The search results are stored in the search is unfolded into corresponding sensory paiiem IDs with 

result holding unit 216. More specifically, ihc associative reference to the image word/sensory pattern correspondence 
IDs are extracted from the unfolded associative word hold- S holding unit 215. For example, sensory pattern IDs "005" 

ing unit 209, and corresponding associative data are and "006" corresponding to image word "refreshing" — 

extracted with reference to the associative word dictionary associative word "forest" are acquired, and a sensory pattern 

211. Next, the association strengths 2114 of the extracted ID "007" corresponding to image word "refreshing" — 

associative data are extracted, and if a status code indicating associative word "tableland" is acquired, 
an antithetic concept is appended to a given associative ID lo in step S2142, the sets of unfolded sensory pattern IDs 

extracted from the unfolded associative word holding unit and image words/associative words are stored in the 

209, the sign of the association strength is inverted to unfolded sensory pattern holding unit 213. The data storage 

indicate a negative association strength. However, if the example at that time is as shown in FIG. 4. In step S2143, 

association strength is already a negative value, that asso- jji image IDs of images having matching levels larger than 
ciative data is discarded, and the next associative data is 15 zero with reject to the sensory pattern IDs stored in the 

checked. In this manner, the obtained association sttengths unfolded sensory pattern holding unit 213 are acquired. Ihis 

are set in a work memory ASCF (not shown) on the RAM process is implemented by the sensory pattern search pro- 

13. ccssing unit 214, and is done for all the sensory patterns 

Then, an associative perspective correspoodii^ to each stored in the imfolded sensory pattem holding unit 213. 
associative ID is extracted, and is compared with that stored 20 g^^p §2144, sets of acquired sensory pattem IDs, image 

in the search perspective holding unit 203. If the two IDs, and their matching levels are stored in the search result 

perspectives match, a predetermined value a is set in a work holding unit 216. In step S2145, two sets of search results, 

memory VPF (not shown) on the RAM 13. If they do not i.e., the image content word search results using associative 

match, a value axO.l is set in the work memory VPF on the words and sensory pattem search results, which arc stored in 
RAM 13. ^ the search re^t holding unit 216, are integrated into one set 

Finally, the image content word holding unit 219 is of search results on the basis of the search weights stored in 

searched for image content words that match associative the determined weight holding unit 207 with reference to 

words corresponding to the associative IDs. If an image those search results. This process is done by the search result 

contenl word is found, its image ID 21911 is set in the found integration processing unit 217. In this process, a method of 

image ID 2160 in the search result holding unit 216, "1" is obtaining common elements of two sets of search results in 

set in the number 2161 of matched associative words, and units of associative words (ANDing search results), a 

the found associative ID is set in the associative word ID method of calculating integrated matching levels based on 

2 162. Then, a value obtained by multiplying the value in the the weights on the searches, and selecting appropriate search 

work memories ASCF and VPF on the RAM 13 by a results in descending order of integrated matching levels, 

predetermined score p based on associative word matching and the like are available. In this embodiment, the method of 

is stored as an associative matching level in the associative calculating the integrated matching levels will be exempli- 

matching level 2163. If an identical image ED has already fied below. 

been stored, the value of the number 2161 of matdied Let A be the associative matching level of an image that 

associative words is incremented by 1, a new assodalive matches an associative word "forest" stored in the search 

word ID is added to the associative word ID 2162, and the result holding unit 216, B be the sensory matching level of 

calculated associative matching level is added to the stored an image that matches the sensory pattern ID "005" corre- 

associative matching level 2163 to update its value. The spending to the associative word "forest", and wl and w2 

search process using sensory patterns in step S3006 and the (wl4w2»l) be the search weights stored in the determined 

search result integration process in step S3007 will be weight holding unit 207. Then, the integrated matching level 

described in detail below with reference lo FIG. 21. is given by: 

FIG. 21 is a flow chart showing the search process using 
sensory patterns in step S3006 and the search result inte- Integmted«ttU:hu.glovel-wiM*Hr2.i? 

gration process in step S3007 in the embodiment of the 
present invention. 

These processes are controlled in accordance with a Integ«ted maiching level-(H'f^^»»'rfi^'« 

processing program stored in the DISK 14. -j^jg integrated matching levels of all sensory patterns of all 

The user inputs a search request for searching images at associative words are calculated. When one image ID has 

the search request input processing unit 201. The search matching levels larger than zero with respect to a plurality 
request contains one or a plurality of query words, search S5 of sensory pattern IDs, a plurality of integrated matching 

per^ectives, and the like. The query word input in this levels an; obtained for one image. However, in this case, an 

embodiment is an abstract image word that expresses image with the highest integrated matching level is adopted 

impressions of images such as "refreshing", "warm", and the as a search result This process is done for all images 

like. In this embodiment, assume that an image word corresponding to either set of search results larger than zero, 
"refreshing" is stored. «o and images whose integrated matching levels are larger than 

In step S2141, the image word held in the query word a predetermined threshold value X are selected as integrated 

holding unit 202 is unfolded into sensory patterns with search results. 

reference to the image word/sensory pattern correspondence That is, it is checked in step S2146 if the integrated 

holding unit 215. This process is done by the sensory pattern matching level of an image to be processed is larger than the 
unfolding processing unit 212. In this embodiment, the 65 threshold value X. If the integrated matching level is equal 

query word holding unit 202 stores the image word to or smaller than the threshold value X (NO in step S2146), 

"refreshing", the unfolded associative word holding unit 209 the flow returns lo step S2145. On the other hand, if the 
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iniegnted matching level is larger than the threshold vahie sensory pattern detenninatioo processing unit 224. That is, 

X (YES in step S2146), the flow advances to step S2147. In the diromatic image feature amounts oorrespondiog to each 

step S2147, the image ID of the image to be processed is of the sensory patterns acquired in step S220S arc compared 

held in the search result holding unit 216 as a search result. with the image feature amounts extracted in step S2203 to 

It is checked in step S2148 if the next image to be processed s calculate their matching level. In this case, the matching 

sliU remains. If the next image stDl remains (YES in step levels for all sensory patterns stored in the image feature 

S2148), the flow returns to step S2145. On the other hand, amount/sensory pattern correspondence holding unit 223 are 

if no images remain (NO in step S2148), the flow advances calailatcd. TTic matching level is calculated using a scheme 

to step S2149 ^ ^ ^^^^ ^ vector computations, statistic processes, or the like 

In step S2149, the sets of image IDs and their integrated lo "^1'*^ f°*«,2f*?i'** i i u ^ n .k 

, , ■ . , . J .u u u u 1-1- •. In step S2207, the matching levels between all the sensory 

matchmg levels are stored m the search result holding unit p«teras and the image calcuSted in step S2206 are stored ii 

216 thus ending the processmg. the sensory pattern holding unit 220 in correspondence with 

An image registration process for regislermg test images j^^gg ^ij^l image. This example is as has already 

wiU be explained below with reference to FIG. 22. j,een described previously with reference to FIG. 5. 

FIG. 22 is a flow chart showing the image registraUoo i5 as described above, according to this embodiment, since 

process in the embodiment of the present invention. multimedia information can be searched based on associa- 

This process is controlled in accordance with a processing tive words that express the contents pertaining to a query 

program stored in the DISK 14. word indicating desired multimedia information, and the 

In step S2201, the user designates an image to be regis- feanirc amount of multimedia information itself is used, 

tered. The image to be registered is designated from those 20 multimedia information having an inappropriate feature 

stored in an external storage device, an image input device, amount which cannot meet the query word can be accurately 

an image database server connected to this image processing extracted. 

apparatus, or the like (none of them are shown). In this In the above embodiment, image information is used as 

embodiment, assume that images servii^ as test images are the information wanted. As for multimedia information 

stored in advance, and the image to be registered is selected 25 other than image information, the present invention can be 

&om them. applied by executing information feature amount extraction, 

In step S2202, an image ID corresponding to an image file and corre^onding the extracted information feanire amount 

name of the designated image, and various kinds of image to sensory patterns. 

information required for registration are acquired, and are In the above description, the image holding unit 218, 

supplied to the image feature extraction processing unit 221, 30 image content word holding unit 219, and sensory pattern 

The image ID is stored io correspondence with the image file holding unit 220 which undergo a search are allocated on the 

name to manage an image, and is acquired by searching data DISK 14 that builds a single device, but these building 

in the image holding unit 218 using the image &le name. components may be distributed on different devices, and 

Various kinds of image information of the image include processes may be done on the network via the NIC 19. 

pixel values indicating the width and height of an image, the 35 Note that the present invention may be applied to either a 

number of bits per pixel, the image size (in units of bytes), system constituted by a plurality of devices (e.g., a host 

the address of the area where a bitmap image is actually computer, an interface device, a reader, a printer, and the 

stored, and the Uke, for example, when the file format of this like), or an apparatus consisting of a single equipment (e.g., 

image is the bitmap format. Since these pieces of image a copying machine, a facsimile apparatus, or the like), 

information are stored in the header field of the image file, ao The objects of the present invention are also achieved by 

they can be acquired by referring to the header field. Even supplying a storage medium, which records a program code 

when the file format of the image is not the bitmap format of a software program that can implement the functions of 

but JFIF or FlashPix, required information can be similarly the above-mentioned embodiments to the system or 

obtained from the header field of a file. Or the image holding apparatus, and reading out and executing the program code 

unit 218 may store such image information, and the image 45 stored in the storage medium by a computer (or a CPU or 

information may be acquired by referring to the image MPU) of the system or apparatus, 

holding unit 218 upon registration. In this case, the program code itself read out from the 

In step S2203, physical image feature amounts are storage medium implements the functions of the above- 
extracted by analyzing the image information corresponding mentioned embodiments, and the storage medium which 
to the designated image ID. This process is done by the 50 stores the program code constitutes the present invention, 
image feature amount extraction processing unit 221. An As the storage medium for supplying the program code, 
example of this process is as has already been described for example, a floppy disk, hard disk, optical disk, magneto- 
previously with reference to FIG. 16. In step S2204, optical disk, CD-ROM, CD-R, magnetic tape, nonvolatile 
extracted image feature amounts cl to cn are stored in the memory card, ROM, and the like may be used, 
image feature amount holding unit 222 in correspondence 5S The fiinctions of the above-mentioned embodiments may 
with the image ID of that image. This example is as has be implemented not only by executing the readout program 
ah-eady been described previously with reference to FIG. 17. code by the computer but also by some or all of actual 

In step S2205, all sensory pattern IDs stored in the image processing operations executed by an OS (operating system) 

feature amount/sensory paUern correspondence holding unit running on the computer on the basis of an instruction of the 

223, and image feature amounts corresponding to those 60 program code. 

sensory pattems are acquired with reference to the image Furthermore, the functions of the above-mentioned 

feanirc amount/sensory pattern correspondence holding unit embodiments may be implemented by some or all of actual 

223. This example is as has already been described previ- processing operations executed by a CPU or the like 

ously witli reference to FIG. 7. arranged in a function extension board or a function cxten- 

In step S2206, the matching level between the acquired 65 sion imit, which is inserted in or connected to the computer, 

sensory pattern and the image feamre amounts correspond- after the program code read out from the storage medium is 

ing to the image is calculated. This process is done by the written in a memory of the extension board or unit. 
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As many apparently widely dififerent embodimenis of the 
present invention can be made without departing firom the 
spirit and scope thereof, ii is to be understood that the 
invention is not limited to the specific embodiments thereof 
except as defined in the appended claims. 

What is claimed is: 

1. An information search apparattis for managing a plu- 
rality of kinds of multimedia information, and searching the 
managed multimedia information for desired multimedia 
information, comprising: 

an image content word holding unit for storing the mul- 
timedia information, and content words which verbal- 
ize concepts expressed in the multimedia information 
in correspondence with each other; 

an associative word dictionary for storing the content 
words and non-associative words w^cfa do not evoke 
bilateral association with the content words in corre- 
spondence with eadi other; 

input means for inputting a query word; 

first search means for acquiring a non-associative word 
corresponding to the query word input by said input 
means from said associative word dictionary, and 
searching multimedia information on the basis of the 
acquired non-associative word; 

second search means for extracting a feamre amoxmt 
corresponding lo the query word input by said input 
means, and searching multimedia iiiformation on the 
basis of the extracted feature amount; and 

integration means for integrating first and second search 
results obtained by said first and second search means 
on the basis of the first search results. 

2. The apparatus according to claim 1, wherein said 
integration means lowers matching levels of those search 
results of the second search results, which correspond to the 
first search results. 

3. The apparatus according to claim I, wherein said 
integration means deletes those search results of the first 
search results, whid) have matching levels smaller than a 
predetermined value. 

4. The apparatus according to claim 1, wherein said 
associative word dictionary stores associative words which 
are associated with the noo-associaiivc words, and 

said first search means searches multimedia information 
on the basis of the non-associative words, acquires an 
associative word corresponding to the non-associalive 
word from said associative word dictionary, and 
searches multimedia information on the basis of the 
acquired associative words. 

5. The apparatus according to claim 1, wherein said input 
means can also input a search perspective. 

6. The apparatus according to claim 5, further comprising: 
a concept discrimination dictionary for storing index 

words and antithetic concepts corresponding to the 
index words in correspondence with each other, and 
wherein said first search means acquires an index word 
and antithetic concept corresponding to the query word 
from said concept discrimination dictionary on the 
basis of the query word and search perspective input by 
said input means, and acquires a non-associative word 
corresponding to the query word from said associative 
word dictionary on the basis of the acquired index word 
and antithetic concept. 

7. The apparatus according to claim 1, further comprising: 
a holding unit for storing associative words and sensory 

patterns in correspondence with each other, and 
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wherein said second search means acquires a sensory 
pattern corresponding to the associative word from said 
holding unit, and extracts a feature amount of the 
acquired sensory pattern as the feature amount corre- 
sponding to the query word. 

8. The apparatus according to claim 1, wherein the 
muhimedia information is image information. 

9. The apparatus according to claim 8, wherein the feature 
amount includes at least one of color scheme information, 
composition information, and shape information contained 
in the image information. 

10. An information search method for managing a plu- 
rality of kinds of multimedia information, and searching the 
managed multimedia infonnation for desired multimedia 
inform^ion, comprising: 

the storage step of storing on a storage medium an image 
content word holding unit for storing the multimedia 
information, and content words which verbalize con- 
cepts expressed in the multimedia mfonnation in cor- 
respondence with each other, and an associative word 
dictionary for storing the content words and non- 
associative words which do not evoke bilateral asso- 
ciation with the content words in correspondence with 
each other; 

the input step of inputting a query word; 

the first search step of acquiring a non-associative word 
corresponding to the query word input in the input step 
from said associative word dictionary, and searching 
multimedia information on the basis of the acquired 
non-associative word; 

the second seardi step of extracting a feature amount 
corresponding to the query word input in the input step, 
and searching multimedia information on the basis of 
the extracted feature amount; and 

the integration step of integrating first and second search 
results obtained in the first and second seardi steps on 
the basis of the first search results. 

11. The method according to claim 10, wherein the 
integration step includes the step of lowering levels of those 
search results of the second search results, which correspond 
to the first search results. 

12. The method according lo claim 10, wherein the 
integration step includes the step of deleting those search 
results of the first search results, which have matching levels 
smaller than a predetermined value. 

13. The method according to claim 10, wherein said 
associative word dictionary stores associative words which 
are associated with the non-associative words, and 

the first search .step includes the step of searching multi- 
media information on the basis of the non -associative 
words, acquiring an associative word corresponding to 
the non-associative word from said assodative word 
dictionary, and searching multimedia infonnation on 
the basis of the acquired assodative words. 

14. The method according lo claim 10, wherein the input 
step includes the step of allowing to also input a search 
perspective. 

15. The method according to claim 14, wherein the 
storage step also includes the step of storing on said storage 
medium a concept discrimination dictionary for storing 
index words and antithetic concepts corresponding to the 
index words in correspondence with each other, and 

the first search step inchidcs the step of acquiring an index 
word and antithetic concept corresponding to the query 
word from said concept discrimination dictionary on 
the basis of the query word and search perspective 
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input in ihe inpui step, and acquiring a non -associative 
word corresponding to the query word from said asso- 
ciative word dictionary on the basis of the acquired 
index word and antithetic concept. 

16. The method according to claim 10, wherein the 
storage step also inchjdcs the step of storing on said storage 
medium a holding unit for storing associative words and 
sensory patterns in correspondence with each other, and 

the second search step includes the step of acquiring a 
sensory pattern corresponding to the associative word 
from said holding unit, and extracting a feature amount 
of the acquired sensory pattern as the feature amount 
corre^onding to the query word. 

17. The method according to daim 10, wherein the 
multimedia information is image infoimation. 

18. The method according to claim 17, whereia the feature 
amount includes at least one of color scheme inforaiation, 
composition information, and shape information contained 
in the image information. 

19. A computer readable memory which stores a program 
code of an information search process for managing a 
plurality of kinds of multimedia information, and searching 
the managed multimedia information for desired multimedia 
information, comprising: 

a program code of the storage step of storing on a storage 
medium an image content word holding unit for storing 
the multimedia information, and content words which 
verbalize concepts expressed in the multimedia infor- 
mation in correspondence with each other, and an 
associative word dictionary for storing the content 
words and non-associative words which do not evoke 
bilateral association with the content words in corre- 
spondence with eadi other; 

a program code of the input step of inputting a query 
word; 

a program code of the first search step of acquiring a 
non-associative word corresponding lo the query word 
input in the input step from said associative word 
dictionary, and searching multimedia information on 
the basis of the acquired non-associative word; 

a program code of the second search step of extracting a 
feature amount corresponding to the query word input 
in the input step, and searching multimedia information 
on the basis of the extracted feature amount; and 

a program code of the integration step of integrating first 
and second search results obtained in the first and 
second search steps on the basis of the first search 
results. 

20. An information search apparatus for managing a 
plurality of kinds of multimedia information, and searching 
the managed multimedia information for desired multimedia 
information, comprising: 

an image content word holding unit for storing the mul- 
timedia information, and content words which verbal- 
ize concepts expressed in the multimedia information 
in correspondence with each other, 

an associative word dictionary for storing the content 
words, associative words associated with the content 
words, and non-associative words whidi do not evoke 
bilateral association with the content words in corre- 
spondence with eadi other; 

input means for inputting a query word; 

first search means for acquiring an associative word 
corresponding to the query word input by said input 
means from said associative word dictionary, and 
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searching multimedia informadon on the basis of the 

acquired associative word; 
second search means for acquiring a non-associative word 

corresponding to the query word input by said input 
s means, and searching multimedia information on the 

basis of the acquired non-associative word; and 
integration means for integrating first and second seardi 

results obtained by said first and second search means 

on the basis of the first search results. 

21. The apparatus according to claim 20, wherein said 
integration means lowers matching levels of those search 
results of the seccmd search results, wbidi correspond to the 
first search results. 

22. The apparatus according to claim 20. wherein said 
integration means deletes those seardi results of the first 
search results, which have matching levels smaller than a 
predetermined value. 

23. The apparatus according to claim 20, wherein said 
second search means extracts a feature amount correspond- 

^ ing to the non-associalive word, and extracts a feanire 
amount which is irrelevant to the query word. 

24. The apparatus according to claim 20, wherein said 
input means can also input a search perspective. 

25. The apparatus according to claim 24, further com- 
^ prising: 

a concept discrimination dictionary for storing index 
words and antithetic concepts corresponding to the 
index words in correspondence with each other, and 

wherein said first search means acquires an index word 
and antithetic concept corresponding to the query word 
from said concept discrimination dictionary on the 
basis of the query word and search perspective input by 
said input means, and acquires an associative word 
corresponding to the query word from said associative 
word dictionary on the basis of the acquired index word 
and antithetic concept. 

26. The apparatus according to claim 20, further com- 
prising: 

^ a holding unit for storing non-associative words and 
sensory patterns in correspondence with each other, and 
wherein said second search means acquires a sensory 
pattern corresponding lo the non-associative word from 
said holding unit, and extracts a feature amount of the 

45 acquired sensory pattern as the feature amount corre- 
sponding to the non-associative word. 

27. The apparatus according to claim 20, wherein the 
multimedia information is image information. 

2S. The q)paratus according to claim 27, wherein the 
50 feature amount includes at least one of color scheme 
information, composition infonnation, and shape informa- 
tion contained in the image information. 

29. An information search method for managing a plu- 
rality of kinds of multimedia information, and seardiing Ihe 
55 managed multimedia information for desired multimedia 
information, comprising: 
the storage step of storing on a storage medium an image 
content word holding unit for storing the multimedia 
information, and content words which verbalize con- 
60 cepts expressed in the multimedia information in cor- 
respondence with each other, and an associauve word 
dictionary for storing the content words, associative 
words associated with the content words, and noD- 
associativc words whidi do not evoke bilateral asso- 
«5 ciation with the content words in correspondence with 
each other; 
the input step of inputting a query word; 
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the first search step of acquiring an assodalive word 
corrcspoading to the query word input in the input step 
from said associative word dictionary, and searching 
multimedia information on the basis of the acquired 
associative word; 
the second search step of acquiring a oon-associaiive 
word corresponding to the query word input in the 
input step, and searching multimedia information on 
the basis of tbe acqiiired non-associative word; and 
the intcgratioQ step of integrating first and second search 
results obtained in the first and second search steps on 
the basis of the first search results. 
39. The method according to daim 29, wherein the 
integration step includes the step of lowering matching 
levels of those search results of the second search results, 
which correspond to the first search results. 

31. The method according to daim 29, herein the 
integration step includes the step of deleting those search 
results of the first search results, which have matching levels 
smaller than a predetermined value. 

32. The method according to claim 29, wherein the second 
search step includes the step of extracting a feature amount 
corresponding to the non-associative word, and extracting a 
feature amount which is irrelevant to the query word. 

33. The method according to claim 29, wherein the input 
step includes the step of allowing to also input a search 
perfective. 

34. The method according to claim 33, wherein the 
storage step also includes the step of storing on said storage 
medium a concept discrimination dictionary for storing 
index words and antithetic concepts corresponding to the 
index words in correspondence with each other, and 

the first search step includes the step of acquiring an index 
word and antithetic concept corresponding to the query 
word from said concept discrimination dictionary on 
the basis of tbe query word and search perspective 
input in the input step, and acquiring an associative 
word corresponding to the query word from said asso- 
ciative word dictionary on the basis of the acquired 
index word and antithetic concept. 

35. The method according to daim 29, wherein the 
storage step also includes the step of storing on said storage 
medium a holding unit for storing non-associative words and 
sensory patterns in correspondence with each other, and 
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the second search step includes the step of acquiring a 
sensory pattern corresponding to tbe non-associative 
word from said holding unit, and extracting a feature 
amount of the acquired sensory pattern as the feature 
5 amoimt corresponding to the non -associative word. 

36. The method according to claim 29, wherein the 
multimedia information is image information. 

37. The method according to claim 36, wherein the feature 
amount includes at least one of color scheme information, 

10 composition information, and shape information contained 
in the image information. 

38. A computer readable memory which stores a program 
code of an information search process for managing a 
plurality of kinds of multimedia information, and searching 

15 the managed multimedia information for desired multimedia 
information, comprising: 
a program code of the storage step of storing on a storage 
medium an image content word holding imit for storing 
the multimedia information, and content words which 
20 verbalize concepts expressed in the multimedia infor- 
mation in correspondence with each other, and an 
associative word dictionary for storing the content 
words, associative words associated with the content 
words, and non-associative words i^ch do not evoke 
22 bilateral association with the content words in corre- 
spondence with each other; 
a program code of the input step of inputting a query 
word; 

a program code of the first search step of acquiring an 
associative word corresponding to the query word input 
in the input step from said associative word dictionary, 
and searching multimedia information on the basis of 
the acquired associative word; 
35 3 program code of the second search step of acquiring a 
non-associative word corresponding to the query word 
input in the input step, and searching multimedia infor- 
mation on the basis of the acquired non-associative 
word; and 

^ a program code of the integration step of integrating first 
and second search results obtained in tbe first and 
second search steps on the basis of the first search 
results. 
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